<html>
<head><meta charset="utf-8"><title>Completion benchmarks · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html">Completion benchmarks</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="229750292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229750292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Kirill Bulatov <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229750292">(Mar 10 2021 at 21:43)</a>:</h4>
<p>I'm considering options on how to write the actual completion benchmarks.</p>
<p>I'm most interested in the flyimport functionality, but in order to test it with something real, I have to add a few external crates into the project.<br>
Should I use something like <code>//- /main.rs crate:main deps:dep</code> in one big file with big number of items and parse that into <code>TestDb</code>?</p>
<p>And then, how I'm supposed to check the actual completion?<br>
Looks like there's no good way to pass some continuous by-char input into those tests?<br>
What are the alternative, to refactor, expose and directly query the main completion method from the <code>ide_completion/lib.rs</code>?</p>



<a name="229830784"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229830784" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229830784">(Mar 11 2021 at 11:04)</a>:</h4>
<p><span class="user-mention" data-user-id="231447">@Kirill Bulatov</span> have you seen the recent macro/parsing benchmarks?</p>



<a name="229830843"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229830843" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229830843">(Mar 11 2021 at 11:05)</a>:</h4>
<p><a href="https://github.com/rust-analyzer/rust-analyzer/blob/db6364fecc2b0e9a95d9aaece820a86265b4cb4f/crates/syntax/src/tests.rs#L45-L65">https://github.com/rust-analyzer/rust-analyzer/blob/db6364fecc2b0e9a95d9aaece820a86265b4cb4f/crates/syntax/src/tests.rs#L45-L65</a></p>



<a name="229831013"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229831013" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229831013">(Mar 11 2021 at 11:06)</a>:</h4>
<p>this I think is the right structure for the benchmark:</p>
<ul>
<li>a slow test</li>
<li>fixed, independent of std, fixture</li>
<li>timing of the specific function</li>
<li>"hash" computation, to make sure we can't optimize the computation away</li>
</ul>



<a name="229831243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229831243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229831243">(Mar 11 2021 at 11:08)</a>:</h4>
<p>The hard thing here is comming up with a test fixture. My feel is that we should avoid pulling existing crates from crates io, and instead write a generator for pseudo crates. That is, a bit of code that generates text for a tree of nested modules with items, maybe even split across several crates.</p>



<a name="229831319"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229831319" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229831319">(Mar 11 2021 at 11:09)</a>:</h4>
<p>See also <a href="https://github.com/rust-analyzer/rust-analyzer/blob/db6364fecc2b0e9a95d9aaece820a86265b4cb4f/crates/ide/src/syntax_highlighting/tests.rs#L229-L248">https://github.com/rust-analyzer/rust-analyzer/blob/db6364fecc2b0e9a95d9aaece820a86265b4cb4f/crates/ide/src/syntax_highlighting/tests.rs#L229-L248</a> for an example of synthetic generation.</p>



<a name="229834808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229834808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Kirill Bulatov <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229834808">(Mar 11 2021 at 11:40)</a>:</h4>
<p>Yes, I've seen the previous benchmarks and apparently grasped the general concepts, thanks for confirming :)</p>
<p>I'm wondering if completions with their "preload almost everything and then query rapidly" approach have any caveats worth thinking of, but I'll check it out along the way now, I guess.</p>



<a name="229834975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229834975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229834975">(Mar 11 2021 at 11:42)</a>:</h4>
<p>I think you want to compute the crate def map outised of the main timing block</p>



<a name="229835017"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229835017" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229835017">(Mar 11 2021 at 11:42)</a>:</h4>
<p>or maybe just use two timing blocks - one for def map, one for completion per se</p>



<a name="229835058"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229835058" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229835058">(Mar 11 2021 at 11:42)</a>:</h4>
<p>or maybe even better: you do the timling, then you "type" one character, then you do timling again</p>



<a name="229835074"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229835074" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229835074">(Mar 11 2021 at 11:43)</a>:</h4>
<p>year, the last one is I think what we want</p>



<a name="229835369"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/229835369" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Kirill Bulatov <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#229835369">(Mar 11 2021 at 11:45)</a>:</h4>
<p>Don't see any other way around to measure the lookup speed, but good point about the preload separate tests: that is mostly the data generator related work and might make sense to check along the way.</p>



<a name="235569843"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/235569843" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Kirill Bulatov <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#235569843">(Apr 21 2021 at 19:44)</a>:</h4>
<p>A sanity check:</p>
<p>am I getting it right that for emulating a continuous user input, I have to replace <code>$0</code> with <code>${next_symbol}$0</code> in the whole fixture and reparse it with <code>ChangeFixture::parse</code> to get the <code>Change</code> and <code>FilePosition</code>? (former goes into <code>RootDatabase</code> as an update, then the db and the latter goes into <code>ide_completion::completions</code> function)</p>
<p>For a benchmark, I'd like to generate big fixtures and parsing the whole fixture multiple times feels a bit weird.</p>



<a name="235635919"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/235635919" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#235635919">(Apr 22 2021 at 08:00)</a>:</h4>
<p><span class="user-mention" data-user-id="231447">@Kirill Bulatov</span> Hm, I don't think we need to emulate continuous user input</p>



<a name="235635974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/235635974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#235635974">(Apr 22 2021 at 08:01)</a>:</h4>
<p>I think it's enough to just do the following:</p>
<ul>
<li>compute syntax hightlighting for the file</li>
<li>type one char</li>
</ul>



<a name="235636017"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/235636017" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#235636017">(Apr 22 2021 at 08:01)</a>:</h4>
<p>That should be enough to expose analysis-level characteristics.</p>



<a name="235636144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Completion%20benchmarks/near/235636144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Completion.20benchmarks.html#235636144">(Apr 22 2021 at 08:02)</a>:</h4>
<p>This won't cover things like "are we prioritizing edits over completion over hightlighting in the LSP layer", but I don't think we are quite ready for that sort of the benchmark</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>